Search system using user behavior data

ABSTRACT

Context-based user feedback is gathered regarding searches performed on a search mechanism. The search mechanism is monitored for user behavior data regarding an interaction of a user with the search mechanism. The response data provided by the search mechanism is also monitored. Context data (describing the search) and user feedback data (the user&#39;s feedback on the search—either explicit or implicit) are determined. This can be used, for example, to evaluate a search mechanism or to check a relevance model.

FIELD OF THE INVENTION

This invention relates in general to the field of data search andretrieval. More particularly, this invention relates to the collectionand use of user behavior data for data search and retrieval.

BACKGROUND OF THE INVENTION

Data on one or more computer systems may contain data useful for a user.However, the data may be too large for the user to find the data bydirect examination of the data. Additionally, some parts of the datarepository may contain information that is not accessible to the user.In many cases, in order to allow the user useful access to the data, asearch mechanism is provided. The search mechanism allows a user toissue a search request (also termed a search query). The results arethen returned for the user.

For example, a web-based search engine is a search mechanism which maybe used to provide search access to information via a web-based search.The information may be a specific data repository, such as a database orother data collection. The information may also be an agglomeration of anumber of different data repositories. Such a search engine may providesearch access to information available from different informationproviders over a network, such as the Internet.

In a typical usage of a web search engine, the user enters a query,which is a set of search terms related to the type of content for whichthe user is searching. The query is transmitted to the search engine,which attempts to locate “hits”—i.e., content that is available on theInternet and that relates to the terms contained in the query.Generally, the search engine either has a database of web pages that areknown to exist, or communicates with external “providers” who maintainsuch databases; the query is “scored” against items in these databasesto identify the web pages that best match the query. A list of resultsis then generated, and these results are returned to the user's computerfor display by the user's web browser.

Typically, the databases contain information such as: the UniformResource Locators (URLs) of web pages, the titles of the pages,descriptions of the pages, and possibly other textual information aboutthe web pages. The user then reads the results and attempts todetermine, based on the text contained in the results, whether theresults correspond to what the user is looking for. Users may thenattempt to retrieve the entire page correlating to a search result. Inother contexts, search engines present results summarizing the pieces ofdata which may possibly be useful for a user.

The utility of the search engine is directly correlated to the qualityof the results provided. In the best case, the most results presented tothe user are presented in order of utility to the user on the resultpage.

Because the quality of the results is subjective, in order to determinewhat the quality of results are, the user's satisfaction must bedetermined. For example, a user can be allowed to use a search enginefor a period of time and then orally interviewed by an interviewer todetermine the user's satisfaction.

In the prior art, quality of individual web pages has been measured byobtaining explicit feedback from a user. At least one prior art webbrowser has attempted to obtain such explicit feedback from a user. Thisbrowser is described in a paper entitled “Inferring User Interest” byMark Claypool, David Brown, Phong Le, Makoto Waseda in IEEE InternetComputing 5(6): 32-39 (2001). In this browser, different pages aredisplayed by the browser. Whenever the page being displayed by thebrowser is changed, a user evaluation of the page is requested from theuser. User evaluations for a given page are collected, to determinewhether users find that page valuable. In this browser, some implicitfeedback is also maintained regarding each page, including dataregarding the time spent on the page, mouse movements, mouse clicks, andscrolling time.

While this technique does gather user feedback, it has limited utilityin situations in which users may have different needs for a page. Forexample, a user who is looking for information about books written byDouglas Adams may evaluate a page on his book The Hitchhiker's Guide tothe Galaxy and give a high score for utility. However, another user whois looking for information on books about traveling cheaply may evaluatethe same page and give it a low score. Thus the technique described willhave limited utility in the wide variety of situations in whichdifferent users may have different needs, or even where a single usermay have different needs for information at different times. In otherwords, the usefulness of this technique is limited because evaluation ofeach page is completely independent of the context in which the userarrived at the page.

Thus, this technique is not useful for evaluating the quality of asearch engine. In general, this technique is not useful for evaluationswhich are context-based, but only for evaluating the quality ofindividual data items, independent of the context in which a userarrived at the data items.

In view of the foregoing, there is a need for a system and method thatovercomes the drawbacks of the prior art.

SUMMARY OF THE INVENTION

User feedback (explicit and/or implicit) on a search performed on asearch mechanism is collected along with the context for that feedback.This allows context-dependent evaluations of the quality of searchmechanisms.

Context-based user feedback is gathered regarding searches performed ona search mechanism. The search mechanism is monitored for user behaviordata regarding an interaction of a user with the search mechanism. Theresponse data provided by the search mechanism is also monitored.Context data (describing the search) and user feedback data (the user'sfeedback on the search—either explicit or implicit) are determined. Thiscan be used, for example, to evaluate a search mechanism or to check arelevance model.

Other features of the invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofpreferred embodiments, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating theinvention, there is shown in the drawings example constructions of theinvention; however, the invention is not limited to the specific methodsand instrumentalities disclosed. In the drawings:

FIG. 1 is a block diagram of an example computing environment in whichaspects of the invention may be implemented;

FIG. 2 is a block diagram of an example architecture according to oneembodiment of the invention;

FIG. 3 is a diagram of a state machine according to one embodiment ofthe invention; and

FIG. 4 a flow diagram showing the process of gathering context-basedfeedback according to one embodiment of the invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Overview

User feedback (explicit and/or implicit) is collected along with thecontext for that feedback. This allows context-dependent evaluations ofthe quality of search mechanisms. If one search mechanism producesresults which garner better feedback than a second search mechanism, itcan be determined that the first search mechanism is more successful atfinding results for a user.

Additionally, it can be used for other purposes. For example, arelevance model can be used to measure the effectiveness of a searchsystem. A relevance model is a probabilistic model which attempts tomodel the useful of results obtained from a search. However, for such amodel to itself be useful, it must be accurate. Feedback from actualusers can be used to validate a relevance model by showing whether thereis a correlation between the predicted utility of results (from therelevance model) and the actual utility of results (from user feedback).

Other uses may also be found for context-based user feedback monitoring.For example, the success of a sponsored link may be evaluated. If, forexample, a sponsored link is returned as a search result, thedetermination of whether it is considered a high-quality search resultmay be useful for the sponsor of the link or for the search mechanismprovider. Additional uses are also contemplated.

In one embodiment, context-based search mechanism user feedback iscaptured by tracking an event stream and following a state machine inorder to determine the state of the user's interaction with the browser.This allows queries to be presented to the user in order to (1) trackwhat state the interaction is in and (2) request explicit feedback fromthe user. Implicit feedback, such as the amount of time spent on a page,among other implicit feedback items, is also tracked. This feedbackinformation allows a determination to be made of whether a search resultwas useful in the context of the original search. Once this data hasbeen collected, it can be analyzed to determine the quality of thesearch mechanism used. It can also be used to test a relevance model andsee if the predictions generated by the relevance model on the qualityof the results generated match the actual quality of the results asshown in implicit and/or explicit user feedback.

Thus, according to one embodiment of the invention, a means is providedfor collecting implicit end-user behavior within search systems andcorrelating that behavior to user reported satisfaction with aparticular search result and with the overall result set.

Exemplary Computing Environment

FIG. 1 shows an exemplary computing environment in which aspects of theinvention may be implemented. The computing system environment 100 isonly one example of a suitable computing environment and is not intendedto suggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing environment 100 be interpretedas having any dependency or requirement relating to any one orcombination of components illustrated in the exemplary operatingenvironment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, embedded systems, distributedcomputing environments that include any of the above systems or devices,and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network or other data transmission medium. In adistributed computing environment, program modules and other data may belocated in both local and remote computer storage media including memorystorage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The processing unit 120 may representmultiple logical processing units such as those supported on amulti-threaded processor. The system bus 121 may be any of several typesof bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus (also known as Mezzanine bus). The system bus 121may also be implemented as a point-to-point connection, switchingfabric, or the like, among the communicating devices.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CDROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 140 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156, such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 20 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the systembus, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

Architecture

In one embodiment, the invention is implemented in or in conjunctionwith a web browser. A block diagram of an exemplary architecture isshown in FIG. 2. The browser, 200, is a web browser which the userinteracts with. A browser helper object 210 interfaces with the browser200, captures browser events, and send the events to a user behaviortracer 220. The core part of user behavior tracer 220 is state machine225. The states of the state machine 225 represent the state of theuser's interaction with browser 200. For example, one state may beassociated with the user having initiated a search, but not yet receiveda result page. The state machine 225 receives events from user behaviortracer 220, tracks the state of the user's behavior as the user uses thebrowser 200, and collects implicit and explicit user feedback. The userbehavior tracer 220 sends data regarding the search and the feedback tothe data acquisition engine 230 which stores or processes the data.

In one embodiment, a Component Object Model (COM) or similar interfacein browser 200 is used to allow an in-process COM server (or equivalent)to be developed which uses the interface in browser 200. This in-processCOM server or equivalent is the browser helper object 210. The browserhelper object 210 can, in one embodiment, be an in-process COM componentregistered under a certain registry key. When browser 200 is started, itlooks up that registry key and loads the object whose class ID (CLSID)is stored there. (A CLSID is a globally unique identifier thatidentifies a COM class object.)

Browser helper object 210 detects the browser's typical events. In somebrowsers, events occur (are “fired”) when certain situations occur. Thedetection of the event can signal that certain situations have occurredin the browser. For example, such events may include events whichindicate the user's navigation from page to page, such as: (1) an eventwhich fires when a hyper link has been clicked to navigate to adifferent page; (2) an event which fires when the history is used fornavigation to a different page; (3) an event which fires when theaddress bar is used to navigate to a different page; and/or (4) an eventwhich fires when the favorites list is used to navigate to a differentpage.

Additionally, such events may be include events which indicate theuser's use of a displayed page, such as: (5) an event which fires when adocument has been completely loaded and initialized (6) an event whichfires when scrolling is taking place; (7) an event which fires when adocument is printed; (8) an event which fires when a document is addedto the favorites list; (9) an event which fires when the window gainsfocus; (10) an event which fires when the window loses focus; and/or(11) an event which fires when a window has been closed. Other eventsmay also exist and may be detected by the browser helper object 210.Some of these events may not be available in some browsers 200, and somebrowsers 200 may require modification in order to provide functionalityfor detection of these browsers.

In addition to notification of the occurrence of the correspondingactivity in the browser, these events in some cases also includeinformation about the activity. For example, when a hyperlink has beenclicked (event 1) the element which has been clicked is sent as aparameter in the event.

In addition to detecting these events which signal the user's pattern ofnavigation and the user's use of a displayed page, browser helper object210 can also access the browser's menu and toolbar and install hooks tomonitor messages and actions which are not reflected in events. In oneembodiment, these hooks are used instead of the use of events asdetailed above.

Once this information is captured by the browser helper object 210, itis passed to the user behavior tracer 220. The events are used in twoways. First, the event may be stored as or used to calculate implicitfeedback data. Such implicit feedback may include: data regarding thetime spent on the page, mouse movements, mouse clicks, scrolling time,and query text entered. Additionally, implicit feedback may includeevent date and time stamps, result positions on page, link text, resultURL, result click-through, result dwell-time, result printed, resultadded to My Favorites, scrolling activity detected on result, andwhether the result was a sponsored link.

In addition to the use of the data from the browser helper object 210for implicit feedback, the data, in one embodiment, is also used totransition from one state to another or to trigger an event in the statemachine. Events in the state machine include user queries for feedback(such as feedback on a specific result or on a query in general) anduser queries for state (for example, when a modified search may or maynot be part of a new query, the user may be asked whether the userintended to continue the same query or start a new query.)

As stated above, in order to enable context-dependent evaluations ofresults of search mechanisms, context data is tracked. In oneembodiment, this context data includes the state of the search, behaviordata regarding the user's use of the browser 200 and, responses to userqueries as to the intent of the user's behavior. Along with contextdata, user feedback data, either implicit or explicit, is also tracked.In this way, the user feedback can be used to evaluate the quality ofthe search mechanism in the context of the search performed.Additionally, relevance models can be evaluated as described above.

The state machine 225 works to detect the contours of a searchsession—when it starts, when it finishes, and what occurs during thesearch session. State machine 225 tracks the possible states of asearch—when the user has finished the evaluation of the particularresult, when the user is done with a particular query, and whenquestions need to be asked of users, such as what feedback the user hason a particular result item or on the query as a whole. Similar eventswhich are detected by the browser helper object 210 may mean differentthings, depending on what stage of the search session has been arrivedat. In order to track this, in one embodiment, state machine 225 is usedto handle the complexity of search events yet maintain flexibility. Thestate machine design pattern is a common pattern which can beimplemented, for example, as detailed in the popular “Gang of Four” bookentitled Design Patterns: Elements of Reusable Object-Oriented Software(Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, AddisonWesley 1994.) Other implementations of state machine 225 arecontemplated, as are other implementations of the user behavior tracer220 which do not include a state machine.

State Machine

FIG. 3 is a block diagram representing a state machine according to oneembodiment of the invention. The state machine first detects that asearch session has been started (for example, by browsing with browser200 to a page with search capabilities and then entering a search). Thestate machine tracks the progress of one search, which may consist ofone query or a number of queries. Each search corresponds to a searchintent, which may be carried through more than one query. Thedetermination of whether two queries are part of the same search, in oneembodiment, is left to the user through explicit feedback. In otherembodiments, this is determined implicitly, for example by thesimilarities between the user's queries or the similarities in theresults obtained.

When the data from the browser helper object 210 indicates that a searchsession is started, Search Session Start state 300 is entered. Forexample, this may occur when the user browses to a page with searchmechanism capabilities. When and if data indicates that a query has beensubmitted (arrow 305), the Query Start state 310 is entered. The querysubmission to the search mechanism will result in a search result pagebeing created at the search mechanism and downloaded by browser 200(arrow 315). When this occurs, the state is transitioned to the ResultList Page Visit state 320.

At this point, the user is looking at a page of results for the user'squery. From this state, Result List Page Visit state 320, the user maydo different things. For example, if the user is looking at page N of Mresult pages from the search mechanism, the user may decide to go to adifferent result page. If this occurs, the state remains in Result ListPage Visit state 320.

In one embodiment, the possibility of the user having finished thesearch is tracked and the user may be queried if it appears the queryhas ended. For example, the amount of time spent in this state and/orthe number of mouse clicks by the user is tracked, and if a clickmaximum threshold is exceeded or a timeout is encountered, the user isqueried to determine if the user is still searching. If the user is notsearching any more, a switch to an end state such as Query End state 360or Search End state 370 is possible, as is an exit from the statemachine, although these are not explicitly represented in FIG. 3. At anystate, the user may close the browser or open a page unrelated to thesearch, and a switch to such an end state or exiting of the statemachine is possible.

If the search is not timed out or exited, though, the user can select aresult item from the result list to examine. When this browsing to theitem result (arrow 325) occurs, the Result Item Evaluation Start state330 is entered. When the requested item result is downloaded to thebrowser 200 (arrow 335), the Result Item Page Visit state 340 isentered. In this state, the user is examining a result item. While theuser browses the result item, for example, going from a first page to asecond page, the state remains Result Item Page Visit state 340. Whilein this state, again a timeout may result in a query to the user and anexit to an end state.

The user may indicate that that the user is done visiting the resultitem explicitly, if a timeout occurs. Additionally, there are other waysin which the user's actions may indicate that the user is done visitingthe result item. For example, the user may browse back to the resultlist, close the browser, or enter a new page in the address bar. When ithas been determined that the user is done visiting the result item(arrow 345), the Evaluate Result Item state is entered. In this state,the user is queried for explicit feedback on the result item. Forexample, the user may be asked to rate the quality or usefulness of theresult item on a 5 point scale by selecting a radio button on a popupwindow on the user's display.

When this evaluation is over, if the user has browsed back to the resultlist (arrow 353), the state returns to Result List Page Visit state 320.If the user has not, that is an indication that the query has ended(arrow 355) and the Query End state 360 is entered. In this state, inone embodiment, the user is queried for explicit feedback regarding thequality or usefulness of the query results as a whole. If the userindicates that the user wishes to refine the query (for example, eitherexplicitly in response to a query)(arrow 363) the state returns to QueryStart 310. Otherwise, the current search has ended (arrow 365) and theSearch End state 370 is entered. In this state, in one embodiment, theuser is queried for explicit feedback regarding the quality orusefulness of the search as a whole. From here the state machine may beexited or, if a new search is begun, the state machine may be reenteredat Search Session Start state 300.

While transitions are occurring between the various states of the statemachine 225 and explicit feedback is being requested and stored,implicit feedback and other user behavior information is being capturedas well. For example, while the user is in the Result List Page Visitstate 320, the users result list page explore behavior is being stored.

If there is a timeout, in one embodiment, a flag is set for the timeout.This is done because many computer systems allow users to use severalapplications at once. If the user has switched to another application,such as reading email with an email application, and the timeout occurs,the user experience may be negative if a dialog box pops up on thescreen to ask him if he is done with the search. Instead, the flag isset. Then, when the user behavior tracer 220 receives data regarding anevent indicating that the user is again using browser 200 (for example,by receiving notice that a mouse move event has been fired by thebrowser 200) then the user will be prompted for the timeout question.

Other state machines are also contemplated. For example, state machineswhich request different explicit user feedback are we grew the statemachine by adding some more complicated transition cases. The statemachine is able to handle quite complex scenarios, the complexity iswell handled and controlled by the way we designed the state machine.

Data Recording

As discussed above, for each search, three types of data are acquired bythe data acquisition engine 230 through the user behavior tracer 220.

(1) Context Data: This is data concerning the query or queries that theuser used during the search. Context data can include data such as: thestates of the search and associated timing, behavior data regarding theuser's use of the browser 200 and responses to user queries as to theintent of the user's behavior. Generally context data allows userfeedback data to be analyzed in the context of what was occurringleading to the feedback.

(2) Implicit User Feedback Data: This data reflects user behaviorthroughout the search session, such as page browsing, scrolling,clicking, and other user behavior. In one embodiment, timing informationis also stored to describe the user behavior (As detailed above, as wellas being used for implicit user feedback, data regarding user behavioris also used along with other data to drive transitions between thestates in the state machine.) This user behavior data is not necessarilythe raw user behavior data stream sent by the browser helper object 210.In one embodiment, the raw data is interpreted and stored as four typesof implicit user feedback data:

-   -   a) user behavior while visiting a result list page, including        the time spent at that result;    -   b) user behavior while exploring a hyper link on the result list        page, such as a search tip link, including the time spent at        that result;    -   c) user behavior for visiting a result item page, including the        time spent at that result and other actions such as scrolling,        printing or adding the document to the user's favorites;    -   (in one embodiment, the active time spent at the result is        measured in these three cases by subtracting any time in which        the browser has lost user focus, for example when a user        switches to another application)    -   d) Result item ignore records the user behavior for ignoring a        result item, i.e. when the user does not visit a result item        though it appears as part of a result list.

These correspond to some extent to states in the state machine 225, andthe interpretation, in one embodiment, of raw data into interpreted userbehavior data of these types occurs in the states of the state machine,while they are being traversed.

(3) Explicit User Feedback Data: This data is the explicit user feedbackwhich has been requested about the result items the user visited and thequery the user have submitted. For example, regarding a search, a query,or a specific result, the user may be asked, via a dialog box, “Did thisanswer your question?” and allowed to enter a response. As anotherexample, regarding a specific result which the user ignored, the usermay be asked “Why didn't you try this result?” and given choicesincluding “I didn't think this would answer my question.”

In one embodiment, explicit user feedback is requested as shown in thefollowing Table 1, with the description of the situation which willprompt the request for explicit feedback detailed in the Descriptioncolumn and the contents for the dialog interaction presented to the useras shown in the Dialog column with bullets representing radio buttons.

TABLE 1 User Feedback. Description Dialog The user has finished Wouldyou say that: evaluating a result This result answered your questionitem This result somewhat answered your question This result did notanswer your question You did not get a chance to evaluate this result(broken link, foreign language, etc.) Other The user exits a You did notvisit this result because: results list without You didn't think itwould answer your exploring all results question (the dialog will beYou've been to this result before presented about some You didn't lookat it or all of the results Other not visited, one dialog per result forwhich feedback is requested) A query has ended For this query were you:Satisfied with the query Partially satisfied with the query Notsatisfied with the query A search session has For this search were you:ended Satisfied with the search Partially satisfied with the search Notsatisfied with the search

The query data, user behavior data, and explicit user feedback datawhich is collected can be presented for examination, used to validate arelevance model, or used to measure the effectiveness of a searchmechanism.

FIG. 4 is a flow diagram showing the process of gathering context-basedfeedback according to one embodiment of the invention. In step 400, thesearch mechanism is monitored for user behavior data regarding a search.At the same time, in step 410, the search mechanism is monitored forsearch mechanism response data regarding the search. This data is usedto determine context data describing the search and to determine userfeedback data describing the search, steps 420 and 430.

CONCLUSION

It is noted that the foregoing examples have been provided merely forthe purpose of explanation and are in no way to be construed as limitingof the present invention. While the invention has been described withreference to various embodiments, it is understood that the words whichhave been used herein are words of description and illustration, ratherthan words of limitations. Further, although the invention has beendescribed herein with reference to particular means, materials andembodiments, the invention is not intended to be limited to theparticulars disclosed herein; rather, the invention extends to allfunctionally equivalent structures, methods and uses, such as are withinthe scope of the appended claims. Those skilled in the art, having thebenefit of the teachings of this specification, may effect numerousmodifications thereto and changes may be made without departing from thescope and spirit of the invention in its aspects.

1. A method with regard to a user performing a search during a singlesearch session at a search engine by way of a search mechanism, saidmethod comprising: monitoring the search mechanism for user behaviordata regarding interactions between the user and the search mechanismduring the search session thereat, the user behavior data comprisingdata concerning a plurality of events, each event corresponding to anaction of the user at the search mechanism during the search session;monitoring said search mechanism for response data regarding said searchsession, the response data comprising a results list; determiningcontext data describing a search during said search session, the contextdata being derived from the user behavior data and from the responsedata and representing an overall context of the search conducted duringthe search session; and determining user feedback data describing saidsearch, the user feedback data including implicit user feedback derivedfrom the user behavior data and explicit user feedback regarding theusefulness to a user of the response data and the user response to atleast one question, the implicit feedback data comprises time spentreviewing a specific item of the results list, wherein the time spent iscalculated by subtracting any time that a user switched to anotherapplication while reviewing the specific item, and wherein the explicituser feedback data comprises (a) a user rating of the quality orusefulness of the specific item reviewed from the results list and (b)the user response to the at least one question concerning the resultslist as a whole; and performing a context-dependent evaluation of theresults of the search engine acquired during the search session, theevaluation based at least in part on the determined context data and thedetermined user feedback data acquired during the single search session.2. The method of claim 1, where said search mechanism is a web browser.3. The method of claim 2, where each action of the user at the searchmechanism is selected from among entering a search query; said usernavigation to a new page using a hyperlink; said user navigation to anew page using a history list; said user navigation to a new page usingan address bar; said user navigation to a new page using a favoriteslist; user scrolling behavior; user document printing behavior; saiduser adding a document to said favorites list; said user switching focusto a different application; said user switching focus back from adifferent application; and said user closing a window.
 4. The method ofclaim 1 where said method further comprises: tracking, using a statemachine comprising at least two states describing progress through saidsearch, which of said states said search is in.
 5. The method of claim4, where said context data describing said search comprises state dataregarding which of said states were tracked during said search.
 6. Themethod of claim 4 where at least one transition between said states insaid state machines is at least partially dependent on explicit userfeedback.
 7. The method of claim 6 where said context data describingsaid search comprises said explicit user feedback.
 8. The method ofclaim 1 where said context data describing said search comprises userbehavior data.
 9. The method of claim 1 where said user feedback datacomprises explicit user feedback.
 10. The method of claim 1 where saiduser feedback data comprises implicit user feedback based on said userbehavior data.
 11. A computer-readable medium having computer-executableinstructions to perform the method of claim
 1. 12. The method of claim 1wherein the search comprises a number of queries from the user to thesearch engine, each query being followed by a response from the searchengine, the method comprising determining context data that describeseach query of the search, including timing and how the user reacted tothe corresponding response, and performing the context-dependentevaluation of the results of the search engine based on such contextdata that allows corresponding user feedback data to be analyzed in acontext of the search that is performed during the search session. 13.The method of claim 1 comprising determining user feedback data thatdescribes the search, the user feedback data including implicit userfeedback derived from user behavior including browsing, scrolling, andclicking behavior.
 14. The method of claim 1 comprising determining userfeedback data that describes the search, the user feedback dataincluding implicit user feedback including: user behavior while visitinga result list page, including time spent thereat; user behavior whileexploring a hyperlink on the result list page, including time spentthereat; user behavior while visiting a result item page, including thetime spent thereat; and user behavior relating to the user ignoring aresult item of a result list.
 15. The method of claim 1 comprisingdetermining user feedback data that describes the search, the userfeedback data including explicit user feedback by way of a dialog boxopened at the search mechanism of the user.